feat: add kube-vip static Pod in a Nutanix handler #558

dkoshkin · 2024-04-23T22:50:32Z

What problem does this PR solve?:
Stacked on #554

This PR moves the responsibility of adding the control-plan kube-vip to CAREN away from infra providers.
This has a big benefit of being able to keep this implementation and the image version consistent across the different providers.

~~Because CAPX is always requires kube-vip to be installed there are no API changes, but in the future we can add support to disable or change the provider.~~
Added an empty virtualIP{} struct to Nutanix controlPlaneEndpoint.

spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          nutanix:
            controlPlaneEndpoint:
              host: x.x.x.x
              port: 6443
              virtualIP: {}

Which issue(s) this PR fixes:
Fixes https://jira.nutanix.com/browse/D2IQ-100364

How Has This Been Tested?:

New unit tests, and tested manually too:

clusterctl describe cluster $CLUSTER_NAME    
NAME                                                                                              READY  SEVERITY  REASON  SINCE  MESSAGE 
Cluster/dkkonvoy-042324-nutanix-03                                                                True                     76m             
├─ClusterInfrastructure - NutanixCluster/dkkonvoy-042324-nutanix-03-wvv6d                                                                  
├─ControlPlane - KubeadmControlPlane/dkkonvoy-042324-nutanix-03-sv2zb                             True                     76m             
│ └─Machine/dkkonvoy-042324-nutanix-03-sv2zb-ds5mv                                                True                     76m             
│   └─MachineInfrastructure - NutanixMachine/dkkonvoy-042324-nutanix-03-sv2zb-ds5mv                                                        
└─Workers                                                                                                                                  
  └─MachineDeployment/dkkonvoy-042324-nutanix-03-md-0-w6mls                                       True                     74m             
    └─Machine/dkkonvoy-042324-nutanix-03-md-0-w6mls-979mw-c582s                                   True                     74m             
      └─MachineInfrastructure - NutanixMachine/dkkonvoy-042324-nutanix-03-md-0-w6mls-979mw-c582s

Special notes for your reviewer:

pkg/handlers/nutanix/mutation/controlplaneendpoint/inject_test.go

pkg/common/controlplaneendpoint/virtualip/kubevip_test.go

jimmidyson · 2024-04-24T10:53:35Z

Because CAPX always requires kube-vip to be installed

Is this true? Is there no way a customer could deploy a different load balancer for this? Should this be opt-in (but default true in our templates so effectively opt-out)?

dkoshkin · 2024-04-24T16:24:28Z

Because CAPX always requires kube-vip to be installed

Is this true? Is there no way a customer could deploy a different load balancer for this? Should this be opt-in (but default true in our templates so effectively opt-out)?

We just got some feedback from an existing customer that is using an external LB where kube-vip would not be needed.
I will update this PR and put it behind an API that defaults to enabled and kube-vip provider.

hack/examples/bases/nutanix/clusterclass/kustomization.yaml.tmpl

pkg/common/controlplaneendpoint/virtualip/fromreader.go

deepakm-ntnx · 2024-04-24T16:43:17Z

@dkoshkin did this work with k8s 1.29.x?

dkoshkin · 2024-04-24T17:29:33Z

@dkoshkin did this work with k8s 1.29.x?

Good question, the pre/postKubeadmCommands are still in the templates so it should but let me try that.

dkoshkin · 2024-04-25T00:11:11Z

Tested with both Kubernetes v1.28

NAME                                                                                                 READY  SEVERITY  REASON  SINCE  MESSAGE                                                                                       
Cluster/dkkonvoy-042424-nutanix-128-2                                                                True                     31s                                                                                                   
├─ClusterInfrastructure - NutanixCluster/dkkonvoy-042424-nutanix-128-2-bc7z9                                                                                                                                                        
├─ControlPlane - KubeadmControlPlane/dkkonvoy-042424-nutanix-128-2-9mrkj                             True                     31s                                                                                                   
│ └─3 Machines...                                                                                    True                     3m22s  See dkkonvoy-042424-nutanix-128-2-9mrkj-hphdt, dkkonvoy-042424-nutanix-128-2-9mrkj-ktxjs, ...  
└─Workers                                                                                                                                                                                                                           
  └─MachineDeployment/dkkonvoy-042424-nutanix-128-2-md-0-mnm7c                                       True                     2m8s                                                                                                  
    └─Machine/dkkonvoy-042424-nutanix-128-2-md-0-mnm7c-qmwq8-bf7mx                                   True                     2m52s                                                                                                 
      └─MachineInfrastructure - NutanixMachine/dkkonvoy-042424-nutanix-128-2-md-0-mnm7c-qmwq8-bf7mx                                                                                                                                 


NAME                                                   CLUSTER                         NODENAME                                               PROVIDERID                                       PHASE     AGE     VERSION
dkkonvoy-042424-nutanix-128-2-9mrkj-hphdt              dkkonvoy-042424-nutanix-128-2   dkkonvoy-042424-nutanix-128-2-9mrkj-hphdt              nutanix://063a872a-fab1-4121-8692-c6a42cbaf75b   Running   71s     v1.28.7
dkkonvoy-042424-nutanix-128-2-9mrkj-ktxjs              dkkonvoy-042424-nutanix-128-2   dkkonvoy-042424-nutanix-128-2-9mrkj-ktxjs              nutanix://f1453c2b-4199-48e2-bc9b-80a312f34a98   Running   3m47s   v1.28.7
dkkonvoy-042424-nutanix-128-2-9mrkj-pm77z              dkkonvoy-042424-nutanix-128-2   dkkonvoy-042424-nutanix-128-2-9mrkj-pm77z              nutanix://af8d6dd7-8edb-4762-9c32-3de93c0ddb6f   Running   2m27s   v1.28.7
dkkonvoy-042424-nutanix-128-2-md-0-mnm7c-qmwq8-bf7mx   dkkonvoy-042424-nutanix-128-2   dkkonvoy-042424-nutanix-128-2-md-0-mnm7c-qmwq8-bf7mx   nutanix://23cb029f-f783-4c75-be6c-bd37c9bf8f6e   Running   3m50s   v1.28.7


      postKubeadmCommands:
      - echo export KUBECONFIG=/etc/kubernetes/admin.conf >> /root/.bashrc
      - echo "after kubeadm call" > /var/log/postkubeadm.log
      preKubeadmCommands:
      - echo "before kubeadm call" > /var/log/prekubeadm.log
      - hostnamectl set-hostname "{{ ds.meta_data.hostname }}"
      - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
      - echo "127.0.0.1   localhost" >>/etc/hosts
      - echo "127.0.0.1   kubernetes" >>/etc/hosts
      - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >> /etc/hosts
      - /bin/bash /etc/cre/install-kubelet-credential-providers.sh
      - /bin/bash /etc/containerd/restart.sh

And v1.29

NAME                                                                                                 READY  SEVERITY  REASON  SINCE  MESSAGE                                                                                       
Cluster/dkkonvoy-042424-nutanix-129-7                                                                True                     8m43s                                                                                                 
├─ClusterInfrastructure - NutanixCluster/dkkonvoy-042424-nutanix-129-7-pwzt5                                                                                                                                                        
├─ControlPlane - KubeadmControlPlane/dkkonvoy-042424-nutanix-129-7-7mvfv                             True                     8m43s                                                                                                 
│ └─3 Machines...                                                                                    True                     10m    See dkkonvoy-042424-nutanix-129-7-7mvfv-2vj6x, dkkonvoy-042424-nutanix-129-7-7mvfv-b2g4p, ...  
└─Workers                                                                                                                                                                                                                           
  └─MachineDeployment/dkkonvoy-042424-nutanix-129-7-md-0-9wfdz                                       True                     9m16s                                                                                                 
    └─Machine/dkkonvoy-042424-nutanix-129-7-md-0-9wfdz-5mnjp-zf5rd                                   True                     10m                                                                                                   
      └─MachineInfrastructure - NutanixMachine/dkkonvoy-042424-nutanix-129-7-md-0-9wfdz-5mnjp-zf5rd                                                                                                                                 


NAME                                                   CLUSTER                         NODENAME                                               PROVIDERID                                       PHASE     AGE     VERSION
dkkonvoy-042424-nutanix-129-7-7mvfv-2vj6x              dkkonvoy-042424-nutanix-129-7   dkkonvoy-042424-nutanix-129-7-7mvfv-2vj6x              nutanix://ac70a906-76fe-4e5a-b10e-22253abc356d   Running   10m     v1.29.2
dkkonvoy-042424-nutanix-129-7-7mvfv-b2g4p              dkkonvoy-042424-nutanix-129-7   dkkonvoy-042424-nutanix-129-7-7mvfv-b2g4p              nutanix://57d8d55f-fb18-4e2d-9089-b76f1e79e433   Running   11m     v1.29.2
dkkonvoy-042424-nutanix-129-7-7mvfv-m52br              dkkonvoy-042424-nutanix-129-7   dkkonvoy-042424-nutanix-129-7-7mvfv-m52br              nutanix://98320226-0e9b-4c2c-82a8-b327ae8ce00f   Running   9m18s   v1.29.2
dkkonvoy-042424-nutanix-129-7-md-0-9wfdz-5mnjp-zf5rd   dkkonvoy-042424-nutanix-129-7   dkkonvoy-042424-nutanix-129-7-md-0-9wfdz-5mnjp-zf5rd   nutanix://62c16adb-780c-48e6-9473-646e016d2763   Running   12m     v1.29.2

      postKubeadmCommands:
      - echo export KUBECONFIG=/etc/kubernetes/admin.conf >> /root/.bashrc
      - echo "after kubeadm call" > /var/log/postkubeadm.log
      - |-
        if [ -f /run/kubeadm/kubeadm.yaml ]; then
          sed -i 's#path: /etc/kubernetes/super-admin.conf#path: /etc/kubernetes/admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml;
        fi
      preKubeadmCommands:
      - echo "before kubeadm call" > /var/log/prekubeadm.log
      - hostnamectl set-hostname "{{ ds.meta_data.hostname }}"
      - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
      - echo "127.0.0.1   localhost" >>/etc/hosts
      - echo "127.0.0.1   kubernetes" >>/etc/hosts
      - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >> /etc/hosts
      - |-
        if [ -f /run/kubeadm/kubeadm.yaml ]; then
          sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml;
        fi
      - /bin/bash /etc/cre/install-kubelet-credential-providers.sh
      - /bin/bash /etc/containerd/restart.sh

pkg/handlers/generic/mutation/controlplanevirtualip/providers/kubevip.go

dkoshkin · 2024-04-25T00:15:12Z

Thanks @deepakm-ntnx for the 1.29 question, it made me realize that CAREN should also be responsible for adding the commands, done in e278d2b

I did test without this commit and it didn't work because it expects KUBERNETES_VERSION to be set to a value.
I will fix the CAPX templates to be more generic and instead only apply this patch based on the built in variable .builtin.controlPlane.version.

pkg/handlers/options/global.go

.../cluster-api-runtime-extensions-nutanix/templates/kube-vip/manifests/kube-vip-configmap.yaml

hack/examples/bases/nutanix/clusterclass/kustomization.yaml.tmpl

This will be moved into the handler so that it can be reused.

Currently a virtualIPProvider will always be passed in to the handler, but refactoring to use a builder pattern makes the code more flexible.

dkoshkin · 2024-04-25T16:34:41Z

Tested with the latest changes after rebase

NAME                                                                                                 READY  SEVERITY  REASON  SINCE  MESSAGE                                                                                       
Cluster/dkkonvoy-042524-nutanix-128-1                                                                True                     36s                                                                                                   
├─ClusterInfrastructure - NutanixCluster/dkkonvoy-042524-nutanix-128-1-w9grt                                                                                                                                                        
├─ControlPlane - KubeadmControlPlane/dkkonvoy-042524-nutanix-128-1-pxzp4                             True                     36s                                                                                                   
│ └─3 Machines...                                                                                    True                     2m     See dkkonvoy-042524-nutanix-128-1-pxzp4-jpbkp, dkkonvoy-042524-nutanix-128-1-pxzp4-k86kz, ...  
└─Workers                                                                                                                                                                                                                           
  └─MachineDeployment/dkkonvoy-042524-nutanix-128-1-md-0-fwksc                                       True                     106s                                                                                                  
    └─Machine/dkkonvoy-042524-nutanix-128-1-md-0-fwksc-np89s-kb94k                                   True                     2m1s                                                                                                  
      └─MachineInfrastructure - NutanixMachine/dkkonvoy-042524-nutanix-128-1-md-0-fwksc-np89s-kb94k

charts/cluster-api-runtime-extensions-nutanix/defaultclusterclasses/nutanix-cluster-class.yaml

api/v1alpha1/common_types.go

hack/addons/update-kube-vip-manifests.sh

pkg/handlers/generic/mutation/controlplanevirtualip/providers/kubevip.go

pkg/handlers/generic/mutation/controlplanevirtualip/inject.go

The kube-vip repo uses this format

Co-authored-by: Shalin Patel <[email protected]>

pkg/handlers/generic/mutation/controlplanevirtualip/inject.go

api/v1alpha1/common_types.go

@faiq

🤖 I have created a release *beep* *boop* --- ## 0.8.0 (2024-04-29)  ## What's Changed ### Exciting New Features 🎉 * feat: give mutators a clusterGetter function by @faiq in https://github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pull/514it was unintelliga * feat: get default sans via cluster object in patch handler for docker by @faiq in #519 * feat: adds nutanix SANs via patchHandler by @faiq in #522 * feat: nutanix csi driver 3.0 by @faiq in #531 * feat: Add additionalCategories field to Nutanix machine details patch by @dlipovetsky in #525 * feat: support setting Nutanix project on machines by @dkoshkin in #535 * feat: Upgrade to CAPI v1.7.0 by @jimmidyson in #555 * feat: CAPI v1.7.1 by @jimmidyson in #560 * feat: Preserve user-managed fields when applying resources by @dlipovetsky in #556 * feat: Preserve user-managed fields when creating namespace by @dlipovetsky in #557 * feat: Added e2e test for capx cluster by @deepakm-ntnx in #523 * feat: add kube-vip static Pod in a Nutanix handler by @dkoshkin in #558 * feat: AWS CCM for Kubernetes v1.29 by @dkoshkin in #564 ### Fixes 🔧 * fix: updated the capx version used by @deepakm-ntnx in #513 * fix: add omitempty to CCM Credentials struct by @dkoshkin in #524 * fix: Add specific descriptions to Nutanix machine details fields by @dlipovetsky in #532 * refactor: setting ownership references to Nutanix CSI Helm Chart Proxies by @dlipovetsky in #565 ### Other Changes * build: Specify go1.22.2 as toolchain to fix govulncheck issues by @jimmidyson in #517 * build: Add metadata for latest v0.7.0 release by @jimmidyson in #515 * refactor: Consistently import CAPI v1beta1 package as clusterv1 alias by @jimmidyson in #518 * build: Fix image tags in release manifests by @jimmidyson in #516 * test(e2e): Use same versions of providers from module dependencies by @jimmidyson in #521 * build: update aws credentials on kind bootstrap cluster by @supershal in #507 * refactor: standardize the code for getting Helm values by @dkoshkin in #500 * build: Use latest k8s for dev and test management cluster by @jimmidyson in #526 * docs: Add how to release doc by @jimmidyson in #530 * build: adds a .envrc.local file for local development for dotenv by @faiq in #538 * refactor: create storage classes directly instead of using CRS by @faiq in #539 * refactor: Move API to caren.nutanix.com group by @jimmidyson in #534 * build: Add Kubernetes v1.30.0 option for bootstrap and Docker provider by @jimmidyson in #541 * build: create .envrc.e2e file from caren e2e config by @supershal in #540 * build: Only allow patch updates to k8s libs by @jimmidyson in #551 * build: Generate CRD YAML by @jimmidyson in #536 * build: Minor golangci-lint config updates for recent versions by @jimmidyson in #552 * build: generated CRDs yamls by @dkoshkin in #553 * refactor: Use separate types for provider cluster configs by @jimmidyson in #537 * docs: Remove additionalCategories from required fields by @dlipovetsky in #543 * build: Upgrade tooling, notably go to v1.22.2 by @jimmidyson in #561 * refactor: provider an entrypoint to the infra provider meta handlers by @dkoshkin in #554 * test(e2e): Add self-hosted e2e test by @jimmidyson in #439 * build: Bundle k8s.io/* back in with sigs.k8s.io/* dependencies by @jimmidyson in #583 * build: Add envtest setup to e2e envrc by @jimmidyson in #563 ## New Contributors * @deepakm-ntnx made their first contribution in #513 **Full Changelog**: v0.7.0...v0.8.0 --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

dkoshkin requested review from jimmidyson and deepakm-ntnx April 23, 2024 22:50

dkoshkin self-assigned this Apr 23, 2024

github-actions bot added the feature label Apr 23, 2024

github-actions bot reviewed Apr 23, 2024

View reviewed changes

pkg/handlers/nutanix/mutation/controlplaneendpoint/inject_test.go Outdated Show resolved Hide resolved

pkg/handlers/nutanix/mutation/controlplaneendpoint/inject_test.go Outdated Show resolved Hide resolved

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from 5d0725c to b95ddc3 Compare April 24, 2024 00:06

github-actions bot added feature and removed feature labels Apr 24, 2024

jimmidyson reviewed Apr 24, 2024

View reviewed changes

pkg/common/controlplaneendpoint/virtualip/kubevip_test.go Outdated Show resolved Hide resolved

deepakm-ntnx reviewed Apr 24, 2024

View reviewed changes

hack/examples/bases/nutanix/clusterclass/kustomization.yaml.tmpl Show resolved Hide resolved

deepakm-ntnx reviewed Apr 24, 2024

View reviewed changes

pkg/common/controlplaneendpoint/virtualip/fromreader.go Outdated Show resolved Hide resolved

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from be3d179 to a2a4ba7 Compare April 25, 2024 00:08

github-actions bot added feature and removed feature labels Apr 25, 2024

github-actions bot reviewed Apr 25, 2024

View reviewed changes

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from a2a4ba7 to e278d2b Compare April 25, 2024 00:17

dkoshkin requested review from jimmidyson and deepakm-ntnx April 25, 2024 00:22

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from e278d2b to c8a08df Compare April 25, 2024 02:46

dkoshkin commented Apr 25, 2024

View reviewed changes

pkg/handlers/options/global.go Outdated Show resolved Hide resolved

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch 2 times, most recently from 39621fe to 3b62fdd Compare April 25, 2024 03:24

jimmidyson reviewed Apr 25, 2024

View reviewed changes

.../cluster-api-runtime-extensions-nutanix/templates/kube-vip/manifests/kube-vip-configmap.yaml Outdated Show resolved Hide resolved

jimmidyson reviewed Apr 25, 2024

View reviewed changes

hack/examples/bases/nutanix/clusterclass/kustomization.yaml.tmpl Show resolved Hide resolved

dkoshkin and others added 6 commits April 25, 2024 09:23

refactor: remove kube-vip template from Nutanix's KCP

936211f

This will be moved into the handler so that it can be reused.

refactor: add kube-vip static Pod to KCP in the handler

d8e1f8d

refactor: better handle nil virtualIPProvider

b52fa65

Currently a virtualIPProvider will always be passed in to the handler, but refactoring to use a builder pattern makes the code more flexible.

refactor: add VirtualIPSpec API and a handler

891b4a4

fix: kube-vip admin.conf pre/postKubeadmCommands

ba7f694

build: Add update kube-vip manifests script and use kube-vip 0.8.0

d7ef1fd

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from 3b62fdd to d7ef1fd Compare April 25, 2024 16:34

dkoshkin mentioned this pull request Apr 25, 2024

build: Add update kube-vip manifests script and use kube-vip 0.8.0 #562

Closed

dkoshkin requested a review from jimmidyson April 25, 2024 16:59

jimmidyson enabled auto-merge (squash) April 25, 2024 17:04

jimmidyson approved these changes Apr 25, 2024

View reviewed changes

faiq reviewed Apr 26, 2024

View reviewed changes

charts/cluster-api-runtime-extensions-nutanix/defaultclusterclasses/nutanix-cluster-class.yaml Show resolved Hide resolved

faiq reviewed Apr 26, 2024

View reviewed changes

api/v1alpha1/common_types.go Outdated Show resolved Hide resolved

faiq reviewed Apr 26, 2024

View reviewed changes

hack/addons/update-kube-vip-manifests.sh Show resolved Hide resolved

faiq reviewed Apr 26, 2024

View reviewed changes

pkg/handlers/generic/mutation/controlplanevirtualip/providers/kubevip.go Show resolved Hide resolved

supershal reviewed Apr 26, 2024

View reviewed changes

pkg/handlers/generic/mutation/controlplanevirtualip/inject.go Outdated Show resolved Hide resolved

dkoshkin and others added 3 commits April 26, 2024 10:58

fixup! add provider field in ControlPlaneVirtualIPSpec

8f86f92

fix: use KubeVIP capitalization more consistently

98ff981

The kube-vip repo uses this format

fix: copy and paste

e569797

Co-authored-by: Shalin Patel <[email protected]>

github-actions bot reviewed Apr 26, 2024

View reviewed changes

pkg/handlers/generic/mutation/controlplanevirtualip/inject.go Outdated Show resolved Hide resolved

fixup! fixup! fix linter

48e7f72

supershal reviewed Apr 26, 2024

View reviewed changes

api/v1alpha1/common_types.go Show resolved Hide resolved

fixup! refactor: add kube-vip static Pod to KCP in the handler

2bd290d

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from eb5bc77 to 2bd290d Compare April 26, 2024 19:32

supershal approved these changes Apr 26, 2024

View reviewed changes

jimmidyson merged commit cbd401f into main Apr 26, 2024
16 checks passed

jimmidyson deleted the dkoshkin/feat-kube-vip branch April 26, 2024 20:07

github-actions bot mentioned this pull request Apr 26, 2024

release(main): v0.8.0 #527

Merged

feat: add kube-vip static Pod in a Nutanix handler #558

feat: add kube-vip static Pod in a Nutanix handler #558

Uh oh!

Conversation

dkoshkin commented Apr 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jimmidyson commented Apr 24, 2024

Uh oh!

dkoshkin commented Apr 24, 2024

Uh oh!

Uh oh!

Uh oh!

deepakm-ntnx commented Apr 24, 2024

Uh oh!

dkoshkin commented Apr 24, 2024

Uh oh!

dkoshkin commented Apr 25, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkoshkin commented Apr 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkoshkin commented Apr 25, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkoshkin commented Apr 23, 2024 •

edited

Loading

dkoshkin commented Apr 25, 2024 •

edited

Loading